import Since from "@site/src/components/since";
import ThemedImage from "@theme/ThemedImage";

# Submitting Results

## Setting API Credentials

<Since version="v1.8.7" />

<details>
<summary>touca login --help</summary>

```plaintext
usage: touca login [--api-url API_URL]

Set API credentials from the CLI

options:
  --api-url API_URL  URL to Touca server API
```

</details>

You will need your API Key to submit test results to the Touca server. You can
obtain your API Key
[using the Touca web interface](../../server/account-setup#find-your-api-key)
and manually set it into your configuration profile. Alternatively, you can use
`touca login` which automates much of this process.

This command will open a browser window to the Touca server to help you log into
your account, then automatically finds your API Key and stores it into your
configuration profile.

If you are self-hosting Touca server, you should pass the URL to your Touca
Server API using `--api-url`.

## Capturing Files

<Since version="v1.8.0" />

<details>
<summary>touca check --help</summary>

```plaintext
usage: touca check --suite SUITE [--testcase TESTCASE] src

Submit external test output to Touca server

positional arguments:
  src                  path to file or directory to submit

options:
  --suite SUITE        name of the suite to associate with this output
  --testcase TESTCASE  name of the testcase to associate with this output
```

</details>

You can use `touca check` to submit any external file(s) that may have been
generated as part of the test run.

```plaintext
$ touca config set api-key=a66fe9d2-00b7-4f7c-95d9-e1b950d0c906
$ touca config set team=tutorial-509512
$ touca check ./output.file --suite=my-suite
```

Since we did not specify the testcase, `touca check` will infer it from
`./output.file` as `output-file`.

You can also submit an entire directory, in which case, every file would be
treated as a separate testcase. You can pass `--testcase` to submit them all as
part of one testcase.

<ThemedImage
  sources={{
    light:
      "https://touca.io/docs/external/assets/touca-cli-check-many-files-single-testcase.light.gif",
    dark: "https://touca.io/docs/external/assets/touca-cli-check-many-files-single-testcase.dark.gif"
  }}
/>

Another use-case of `touca check` is submitting the standard output of any given
process to treat as a test result.

```plaintext
$ echo "hello" | touca check --suite=my-suite --testcase=my-testcase
```

Note that in the above case, setting `--testcase` was mandatory since there is
no filename to infer it from.

## Running Tests Locally

<details>
<summary>touca test --help</summary>

```plaintext
usage: touca test [-h] ...

Run your Touca tests

options:
  -h, --help            show this help message and exit
  --testdir TESTDIR     path to regression tests directory
  --api-key API_KEY     Touca API Key
  --api-url API_URL     Touca API URL
  --team TEAM           Slug of team to which test results belong
  --suite SUITE         Slug of suite to which test results belong
  --revision VERSION    Version of the code under test
  --offline             Disables all communications with the Touca server
  --save-as-binary      Save a copy of test results on local filesystem in binary format
  --save-as-json        Save a copy of test results on local filesystem in JSON format
  --output-directory OUTPUT_DIRECTORY
                        Path to a local directory to store result files
  --overwrite           Overwrite result directory for testcase if it already exists
  --testcase TESTCASES [TESTCASES ...], --testcases TESTCASES [TESTCASES ...]
                        One or more testcases to feed to the workflow
  --filter WORKFLOW_FILTER
                        Name of the workflow to run
  --log-level {debug,info,warn}
                        Level of detail with which events are logged
  --no-color            Do not use color in standard output
  --config-file CONFIG_FILE
                        Path to a configuration file
```

</details>

Touca CLI makes it convenient to locally execute Touca tests written using the
Touca Python SDK. Simply navigate to any directory and run `touca test` with
your preferred options to execute all the Touca workflows in that directory.

```plaintext
$ git clone git@github.com:trytouca/trytouca.git
$ cd trytouca/examples/python/02_python_main_api/
$ touca config set api-key=a66fe9d2-00b7-4f7c-95d9-e1b950d0c906
$ touca config set team=tutorial-509512
$ touca test
```

<ThemedImage
  sources={{
    light: "https://touca.io/docs/external/assets/touca-cli-test.light.gif",
    dark: "https://touca.io/docs/external/assets/touca-cli-test.dark.gif"
  }}
/>

## Running Tests on a Test Machine

<details>
<summary>touca run --help</summary>

```plaintext
usage: touca run -p PROFILE [-r REVISION]

Run tests on a dedicated test server

options:
  -p PROFILE, --profile PROFILE
                        path to the profile file
  -r REVISION, --revision REVISION
                        specific version to run
```

</details>

`touca test` is convenient for running Touca tests on your own machine or as
part of the CI workflow. But these two environments have important limitations:

- Running Touca tests at scale with a large set of test cases could be time
  consuming, leading to longer build times.
- Our test workflows may require access to input data that could be large in
  size and difficult to provision on a build server.
- Performance benchmarks obtained on a build server are prone to noise.

To allow testing mission-critical software workflows at scale, Touca CLI
provides the `touca run` command that is designed to be used on a dedicated test
machine. This command takes a recipe describing your software delivery pipeline
specification to download and deploy new releases of your software and to run
your tests against it.

Let's imagine that we want to automatically run Touca tests that are packaged
and deployed to JFrog Artifactory as part of our build pipeline. Our objective
is to setup a testing pipeline that periodically checks Artifactory for new
versions of our artifact, pulls the new version, installs it, executes the Touca
tests, and archives the generated test results.

```bash
touca run --profile ./sample.json --revision 1.2.3
```

Where `sample.json` is a test profile that describes our test setup:

```json title="sample.json"
{
  "artifactory": {
    "artifact_url": "https://touca.jfrog.io/artifactory/{repo}/{group}/{name}/{name}-{version}.tar.gz",
    "repo": "example-repo-local",
    "group": "io.touca",
    "name": "sample_app"
  },
  "install": {
    "installer": "sample_app/install.sh",
    "destination": "./deployed"
  },
  "execution": {
    "executable": "sample_app",
    "output-directory": "./output",
    "suite": "sample_app"
  },
  "archive": {
    "dir": "./archive"
  }
}
```

Check out our
[JSON schema](https://github.com/trytouca/trytouca/blob/main/sdk/python/touca/cli/profiles/profile.schema.json)
file for a full list of supported fields and scenarios.

We can use Linux Cron Jobs or Windows Task Scheduler to automate the execution
of the above command. If we intend to run Touca test for every version of our
code, we can use a short polling interval such as 5 minutes. The CLI gracefully
exits if there are no new versions of our software available in Artifactory.

:::info

For Touca enterprise customers, we provide the test infrastructure and tooling
required to remotely execute Touca tests at scale and based on custom
requirements.

:::

## Submission Mode

<Since version="v1.8.7" />

To provide quick feedback, Touca CLI commands `touca check` and `touca test` as
well as the test runners for all SDKs run in synchronous mode by default. In
this mode, as test cases are executed, the test output will include whether the
test results submitted for a given test case match the baseline.

If you prefer to submit the captured results for a test without waiting for the
comparison results to complete, set the configuration option `submit_async`.

<ThemedImage
  sources={{
    light: "https://touca.io/docs/external/assets/touca-cli-async.light.gif",
    dark: "https://touca.io/docs/external/assets/touca-cli-async.dark.gif"
  }}
/>
